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J m Abstract- 

A new system of order .analysis , developed by Take^ ^and called Item 

Relations Structure. Analysis (IRSA), was described >nd ps&d^for 

examining the structural* relations among a set of 24 items- on the ■ 

addition and subtraction of fractions. A digraph showing 16 chains of 

items that had discernibly common features was generated by this method, 

* . • * «i 

and implications'' for ^diagnostic error analysis vafere discussed. » , 
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Introduction s a 

Tatsuoka '& Tatsuoka (1981) demonstrated^ in tbeir study fhat even a 

crjLtefion-r$ferenced test, in w'hich items are chosen from a single 

content domain, violates a homogeneity of the test items when students 

» , » * 

use a variety of different methods to solve the* problems . Thus, 

. examining the underlying cognitive processes that are adopted by the 

students is very important before any test theory models, such as Item 

Response Theory or criterion-referenced test theory', are applied for 

analyzing the pferf ormancj&s of the student^ on th£ tests. These mbdern 

test theories Require some strict assumptions on the , structure of an 

(item domain, although >they are very useful in many way^. 

Investigating the structure of test, items can be doner by several 
different' procedures — factor analysis, scalogram analysis (Guttman, 
1950) Loevingei^s (1948) measure of test homogeneity and or<jer analysis 
(Krus, 1975 ^ Cliff, 1977). ^Unfortunately , these procedures have failed 
to produce satisfactory- results fr^om achievement data obtained from a 
series of experimental' studies (Ta,tsuoka & Birenbaum, 1979, 1981; 
Birenbaum & Tatsuoka,' 1981; Tatsuok^ & Tatsuoka, 1980). 

Order analysis has beea'used "in constructing- a hierarchical 
structure of the items and instructional units (Airasian L Bart, 1973; 
Bafrt,& Krus, 1973). Wise (1981) has developed a new order-analysis 
procedure to extract unidimensional subsets from the total' set of test 
items and Tak|y£t (1981) has defined a new order structure by using the 
expected proportion? of dominance relationships between two items. 
Takeya's order structure is mathematically „ elegant , and it has algebraic 
relations with Loevinger's homogeneity ift'dex, Mokken's index (Mokken, 
1971), caution in4ex r (Sato„ 1975.) and Cliff's index C t 3. 

• Therefore, wfc will adopt Takeya's, order analysis (called Item 
Relations Structure Analysis, IRSA) to^ examine the item relationship of 
fraction problems (Klein,' et al.,\1981). The advantage of using IRSA-4s 
(according to Takdya) that it enables us fco^ sfee a cognitive aspect of 

9 



the students' performances on the % items.to a cerpain extent. Since it 
generates a digraph representing the hierarchical structure of the items, 
ijfc will — at the very least — allow us to* check' the extent to which we 
have succeeded in constructing problems tbj^t require a hierarchically 



specified set of skills for solving them. ' Thus, with some refinement* 
and* further development, it should enable us to construct tests for 
diagnostic purposes with built-in remedial tasks. 



A Brief Summary qf the Theoretical 
v Background of Order Analysis 



I 



In the traditional scoring of an n-item test, each item receives a 
scorfe of T when the response to the item matches the key 'and 0 when it ^ 
does not. For each subject 1^ -the resulting -response .pattern can be 
represented by a row vector: ' 0 

2k." (xklf-» *kj>'-> x kn>5 
t *kj - 1.0; k-l..., N; j»l,...n. * 

,Mokken (1971) defines "a perfect scale" by describing it as a step 
function: P(*kj - 1 Je), - 0 if subject k's ability level ,6k is located 
to the left of the difficulty level, d^'of item j on the horizontal axis 
.and" 1 if the 6k is to the right of dj. The following figure illustrates 

the perfect-scale function. . 

& 1 

^ Insert Figure 1 about|here 

v If ail subjects perform on item j according to ,the Mokken's # 

perfect-scale function, then item j is s^id to be a perfect , item. , 

However, real data usually -don' t follow this logic and quite a number of 

Subjects score, 0 even if B 4 > dj. A contingency table of two' items, item 

* 

i with difficulty dj[ less than that of another i'ten>, dj, wtil explain 
the^situation. Let 'the^ items be arranged in ascending order of 
difficulty, so that d± < dj;* when i < j\ (Lower-tlum^ered items are . 
easier than higher-numbered items.) * , 4 

\ item, j 
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Figure i.^-Mokken's Perfect-Scale Function: PCx^ ■ l|8) = 0 if 6 < 

. ' ' * / 

' = 1 if e >' d 
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If items i and j are perfect then noi =.0. W call the response 
pattern (0,1^), wittf item i wrong but item j .right, "inconsistent." For 
a set 9f N subjects, if the inconsistent cell '(0,1) is empj:y (i.e., if 
,noi ■ 0) then' the set of N two-dimensional response, vectors forms a 
Guttman scale., • v - 

A number of researchers have, tried to cope with the problem of 
inconsistent responses*' and have developed varioiis "types of measures (or 
used preexisting ones) « the proximity Qf two- items, agreement), 
coefficient, <J> coefficient,, 4ta coefficient, tefrachoric correlation, i 
etc. Thdse measures express some aspects gf the relationship between 1 
the two items; but not all aspects. 

Takeya's index r^j is based on' the dominance relation between 
items i and j and considers the statistical independence or dependence 
of * the scopes obtained from the two items. Now, let us denofre a colurari 
vector of a given data ^matrix (X kJ ), k-l,...,N; j»l,...,n 
by §j and the complement of §j by . 1 

Then the proportions of rights and wrongs, respectively, for item j a're 
expressed by ' ■ • 

P(§j)- - (l/N) jx k j , ■ 



and 



k-i ' 

« / 



P(fij) " (1/N)^2p - x k j) 



- -1 - P(6j). • • 

The proportion P^, 6j) of subjects getting both items i and j 
right is given by , 



P<ei, 6j) - - n n /N 

The proportion of subjects getting .item" i wrong* and item j right is. 
P(5i 6j) - U/wj^O " x kl )x kj . noi/N 



J 



Similarly, 



» (i/N) j ^(1 - = «io /N *y» 



P(§i, Sj) - d/N^d - - x^) = n 00 /N 

These proportions are summarized in the following contingency tatle of 

\Ke items i and j.* , w : 

Insert' Figure 2 about v here 
The varianc^, covariance ^nd correlatioh of the four vectors ' 

JSi.^jSji lii Ij are S* ven below: 

* d 2 OjX - POj) P(9j) = P(9j)[l - P(9j)] , /, ' 

* aOi, 6j) - P(6 1> 9j) - P(6i) POj) ; q 

* p(»i, 6j) 23 [POJ 6j) - P(e 1 )P(e j )]/J P(a i )p(e)P(e.-)P(0 j ) 4 

^ Symbolizing item i as 0-^ (for "observable No. i"), the definition 

of Takeya's^jrelation > between 0^ and Oj is that 

0i > 0j iff P<5i, 9j) < pP(9i) PX9j) , - 

where p is d constant between 0 and' 1 , and usually 0.4 < p < o.6. f 
His relation is based on a criterion of "negative dependence" between 
'vectors §i and 9j. If the proportion in cell (0,1) — 0£ wrong and 0j 
rigtit — is less than about one-halfw of what would.be expected when 

responses to items i and j are independent, then 0± > Oj. He further 

defines " the following:* - * • ' "i 

•/ Dl: 0 t > Oj and Oj -^y> Oj <=> 0± ~ Oj < 

1 . (items i and j ar6 "equivalent") 

D2: 0 ± > Oj and Oj -X-> Oj <«> 0 ± Oj y 

(item if is '"easier" than item j)^ 
D3:* Oi 0/j and Oj ---> Oi O^Oj -»,0i 

(item, j is "easier" than item i) 
* D4: 4 Oi -3F> Oj and Oj -X-> Oj <»> Oi X 0 jv f 

(item i is orthogonal to item j) J 
for the sake o'f mathematical convenience," he denoted the complement 
of P(6i, ej)/P x (6i) P(6j) 'by r *j ' 1 

and named it the Coefficient of_ Oyciinality : / 
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Figure 2: The Contin§encyNTab*Le of Items i and j 
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•rfj - 1 -.P(6i„ 6j>/P(ei). PC9j) 
The Coefficient .of Ordinality r^j can be' rewritten as the^ratio of 
(J coefficient and the quantity-' f 

[P(8i) P(6i)/P(ei) P(6j)] 1/2 * • 



U) r 4j = pOi, 8j) /4 P(6i) P(ej)/f(ei)-p(Bj) . 

Coefficient of Ordinality r*j enables us to see -the difficulties of items 

i and j as well as how the, b coefficient relates' interactively to th^ 

prder, of the tvo items. K^us & Bart (1973) and' Bart & Afrasian (1975) 

i ' - 

defined the dominance relatioiT of two Items, by taking the probability, 

P(6j[/6j) into account. Item i is said to be dominated by -item j if 

P(6jL> 9-j) ^ c for some small number z , say 0:02 { e { o.o4-. Bart,, 

et al/s dominance relation does 'not refle-ct either p(6£, 6j) or 

P(6i, *8j), /Thus, 'the dominance gelations of items having a -high i 

(J 'value with similar difficulty levels^often cannot be defined. Also,^ > 

two litems that are independent can have a consistent dominance relation 

if their 'difficulty levels .a$e clearly different; s 

T^ike^a's definition of r*j enables us to avoid these conflicts. 

Re proved a series of properties Regarding the relations among item 

difficulty, P(6^, 6-j ) ai)d dominance of items. They are listed below 

without proofs'. " 

Property 1: If Oi -> Oj (by D2) then p (6^ , 6j) > 0 and P(9i) >J?C$]). 
Property 2: If P (6^ 6j) > Q and P(8i) > P(6j) then r*j > 
Property*3: ' A set of Items whose elements satisfy the circularity 

°±l ~> Pi£> 0 ifr -> °i3 ,### »^im 0jL l doe ^ not exist# 
Property A: If. P(6i) £ P(Bj) and 0j[ > Oj then 0j[ - Oj (DA). 

Property 5: If 0^ ~ Oj, thenp(6i, 9j) > 1 - p. * 

A matrix called .the Item Relation Structure Analyses Matrix 
(IRSA Matrix) is formed by calculating r^j for all pairs of i and j / 
and i£ rfj is* larger tjtan a constant, say .60, replacing the (i,"j)-cell by 

1, otherwise by 0« s 

Examination of Item Structure of Fraction Test 
Item Relation Structure** Analysis Matrix: IRSA Matrix 

We have constructed tests of 48 fraction, additi/on and 42 
subtraction problems which are expected to be capable of diagrfosing 
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erroneous rules resulting from ^incomplete knp^ledge or misconceptions 

occurring at one or more components ii^ the- procedural trep givgn in 

Figure 3. ^ A detailed description is given in "ReseaJch^Report 81-6 
* *Wt * 

(Klein, *et al. , 1*9 8 1 ) • The two test^ consist of Iwo parallel subtests 

^# . * * 

in which one of each, pair of purportedly parallel items is placed in 

each .subtest. It was noted tha't constructing two parallel items in 

t£rms of having*"Identical procedural steps (number of reducing .needed to 

get th.e, right answer, or obtaining the .least Qomman' denominator of the 

two fractionsby*a ptlrae-f actor approach) requires great care and 

attention* A, couple of, hjindred bu^s discovered in our prev!ous\^studies 

of signed~nun;ber arithmetic (Tatsuoka, et oil. , 1980; Tatsuoka & 

TaiJboka, 198L; Birenbpum & Tatsuoka 1980, 1981) have^shQwn the 

ftecesalt^'of |peciaf^attention*^to the thorough examination of detailed 

"procedural steps*" / 

"1^** m \ * ' Insert Figure 3 about here * 

I 7 * 

The' conventional, item analysis using th<f parlances and % 

correlational- relations of items fhd the. total scores will not work £pr 

test^ aim^d at diagnostic -use. 'Examination df it^ga performances with- 

respect to the expected operational functioning of each item when they 

were constructed must be carried out into the !Level of individualized 

behavfbr of each item and the interactions among items. IR&A method 

seems to provide the information of item behaviors with the^help of a 

digraph representation of -the dominance relations among items. 

Mo'r^over K TaK^t^g^r* j has algebraiq' relations with traditional 

statistics such as reliability, \:iiff ^"consistency index, Loevengier's 

homogeneity inde^ (Takeya, 1981). 

^ We adopted IRSA to investigate the fraction test items. Table 1 is 

the Item Relatioji Structure Matrix in which the 24-items from the fifcst 

subtest of the 48-item addition test were sorted by their p-values' 

(proportion cortect)/ " * / * 

Insert Table 1 about here 

Table 2 provides the p-^values of the' 24 items. 



Insert Table 2l&bout here 
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METHOD B 



*SEE FIGURES 3,^,38 



FIGURE? 3 .' A Pro^edurol Network for Adding Two Fractions 

• ' . 9 - . . 



y ' . Table 1 

Item Relation Structure Analysis, (ItlSA) Matrix of the* 
• * % First 24 Fraction Addition Problems ' c ' 
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.Difficulties, P(6.) of the First 24 
Fraction Addition "Problems 
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The scoring of the; test was -obtained by matching each part of an 
answer to the corresponding part of, the right answer. By "parts" is 
here meant the whole number part (W) 'of the answer ,' and the denominator'- 
and numerator Qf the fractional part (F)« For example, if the right 
answer requires reducing the fraction to its lowest terns, and, a student 
did not carijy out the reducing but go^ the whole number part right, then 
a three-element binary vector (I 0 0) 13 stored as the student's score 
in *the PLATO system. The item difficulties shown in Table 2 are based 
on the customary right/wrong score multiplied b^ th^ three dements of 
the, score vector such as (1 0 0)'.* 

In oi^ter* to construct a digraphed tr^f from- the IRSA matrix, we 
must first extract useful, systematic information existing 4mong various 
subsets of items* As v it stands, the IRSA matrix in Table 1 is too * 



complicated to' extract chains because' there are too maqy items, and the 
dimensionality (in thfe Guttman-scale s^Qse) of^the dataset is large — 
even though, in principle, ^±t may be ufffcd like an adjacency matrfx, with 
one exception as noted below. ^ 

Super-Order of Items % A 

Takeya's item relation " >" does not satisfy the transitivity law 

which- is crucial for extracting a chain from the original set of items. * 
Although he extended his theory to the else of' thtes or more ite\s and 
defined the'epef f icient of ,ordinaiity for a finite number of items, 
transitivity still does>not necessarily hold unless inequality (2) is 
satisfied. 'That is, 

" 0j[ > 0j and 0 j > 0^ imply 0j[ > provided 

(2)' P(B U %l) KPfru ej)P(e^ &)/<!- P) i 

It can be seen fn real data that inequality (2) .sometimes does not hold. 
To cope with this'problem we proceed as fellows* [Takejra iq not 
explicit about the actual algorithm 4 he uses to extract his chains, whichr 
is presumably a trade secret of* his company. What we describe below may 
therefore not -be the most efficient algorithm, btfr.it does work.) 

VP * * 

For each item 0±, we define its antecedent det .A± as the set of all 

• A t 

items, 0j that have" the relation 0j > Oj. (The members of are 

* ■ * 

* % 12 ^ J 



tho'se items Oj in the IRSA matrix that have unit entries in the column , 

representing 0±*) Next, for eachritem Oi, in the order in whictrthey 

occur from top to bottom in the IRSA matrix, we list -the item numbers m» 

of those itams O m that occur to the right of and whose 4ntecedent ^set 

Insert Table 3 about here 

A m is a subset of Aj. Table 3 sjiows such a listing for the present 

** • * * 

example. Next, we reconstrue tjiis listing to give new, formal 

definitions .of A^ as sets of the integers standing in the line traded by 

the subscript i of Aj* Thus-, for example, the fourth ty>w of Table 3, 

headed by 24, is ^construed as the formal redefinition #24 3 U5, 16, **7, 

10}. [We v realize that^the numerals in each cow, themselves originally 

stood for antecedent sets with those numerals as subscripts. It may 

therefore seem» that we are talking about "sets of sets that are subsets 

of the' former" *in a manner reminiscent; of Russell's paradox! However, we 

are merely making formal redefinitions of each Aj[ for the sole purpose 

of facilitating the description of the next and fifial step. t We ace npt 

asserting, for example, that A£4 * {A'15, A]^, A17, A^q} anc * "cStamitting 

the fallacy of confusing two. levels of "set-hood". Nor are'w^making 

any *conf licting' redef iti^jt ion like A£4 " (0l5> O16, 0i7,^ 0iq}> which 

contradicts the original (and "true) definition of A24. finally, 

starting fi^om the bottom of Table 3 we trace sequences, Aj^t***) 

A^ , of antecedent sets that successively subsutae ^he ones to the left; 

s , - * 

i.e., Aj[ 0 c • • * c &i • In the present example,- the 16 sequences 4 
shown below are found. (Note that* five of the lines have two-character 
headings like c,9". Each of these lines lists two sequences differing 
ohly In whether the last set is k 2 or £5, as d$ the sequences in lines a, ' 
and . ) % * 

a, 'a" A10 c A17 c A16 <= A15 c Ag c Aig c= A3 e An c k 2 (or A5) 

b A10 c A17 c Ai^ <= A15 c Ai3 c A4 c A2 . \^ 

c> c ' Axo c A17 cz Ai6 c A15 h k 2 0 c A ll = A 2 ( or A 5) * 

d, d" A 10 c A17 c Ai2,c A7 c A\q c A3 c An c A£ (or A5) 

«e, e" ,Aio c A17 c A19 c An-cz A2 (6r A5) ( * 

• - •• 13 • IS" ' * 



' • Tabled / <- 

-Antecedent Sets A^ Construed as Sets of Integers m 

Such That A , c A 
< 1 m 
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f , . kio c A17 c Ag c A2 «<br A$) 



MO. <= A*? c A 9 c A 5 * 



A 10 c A 17 c A44 6rAi 



o, o' A A^o v cAi7 PA*23 cA^ c A2 (or A5) 



A'2icA22<= ; A2 



TJhe subscripts of the A's' in: each of these sequences identify the items 
that constitute a chain of approximately, ukldimensional Hems. What we 
have accomplished 4>y the set °f operations described above is to 
identify subsets of items, for which transitivity of th& relation 
'holds. 

Appendix I presents the '48-item test administered to 154 local 

junior-high students. Item No's. 21 and 22 in the first subtest (the, 

first 24 items) involve the addition of a mixed number and a whoje 

number; these items are in chain. r~ It is obvious that W + F, W*+ M 

types usually don't involve the operation of finding the least' common 

denominator, so /they form a ch^in independent from other types of skills 

unless a student uses Method A (convert to a F + F type). A + M type 

items (13, 4, 12^-6, 9, 1) are not included in the longest chains, a^and 

a', in whiph the types of items are mostly F + F. Simplifying a 

fraction to a mixed number, converting a mixed number to a fraction seey 

to cause children a coasiderable amount of trouble. Items £ and 5 

involve relatively prime numbers in their denominators as well, as/the 

simplification of the final answer. The % shaded nodes in the digraph of 

Figure 4 sta^d for' the items whose final answer needs simplification. 

It is fairly v clear that if s student has some mfseopception about the 

simplication procedure, then he/she tetidp to repeat the same mistake 
« * * * 

until it is corrected* Thus, the items requiring simplification tend to 

be located toward the end of the chains, in Figure 4." The items that' are 

Insert Figure 4 about here 

Similar types such as W + M or W + F — adding a whole number and'a 

mixed (or fraction) — tend to clusfeer together,' but 21 end 22 which are 

of the M + W*type don't haye any delation with other, types of items. 
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Items 20 and 23 behave differently from 21 and 22* More extensive .errgj* 

\ * 
analyses should provide a resolution of such apparent anomalies. 

/ Summary and Discussion' 

have described innovative techniques that will help, in * 

constructing error-diagnostic tests and examining the appropriateness 

and necessity of eacl^ item in order to enable the test to provide a 

specific description of misconceptions. Brown & Burtort's (1978) 



artificial intelligence approach (B0GGY) is innovative and very 
impressivfe. However, as the numbet of discovered bugs increases 
' enormously, the algorithm of a computet progr^ becomes more and more 
complicated.* Apart from their approach, the authors (T*at*suoka, et al . , 
1980; Tatsuoka & Tatsuoka, 1980; • Tatsuoka & Linn, 1981) have been 
- workfhg to develop psychometric models that should have capabilities 
comparable to BUGGY and, moreover, will be able to handle several 
hundred bugs >with a probabilistic 4PP roach « 

A technique to investigate the item structure with respect to the 
roles of each iteto in determining the student's- misconceptions will be 

A m * 

in great demand for error-diagnostic testing. Itedj^Relation StructufceM 
Analysis was tried out to obtain .our much needed information, and a new 
chain-formation procedure was introduced in this paper. The result toore 
or less confirmed that our item-construction procedure was on the r^ht 
track, but further and more extensive investigations from different ~ 
angles — such as the examination of whether two purportedly parallel 
* items are really parallel — will b£ needed. 
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Appendix I 



The First 24 -Items in a- 48-Item Addition 



Problems Xn Fractions Test 



/ 



\ 



X 



ERLC 



25 



- ' f ■ . • name* • " " . 

. / GRADE • 
tEACHER 

'■ -V ■ / 

N SHOW ALL YOUR WORK SO THAT YOU CAN RECIEVE PARTIAL CREDIT. , 




\ 

• / 

• 

t 

* 


' 2 2 i 12 - 

\ 


7. 2»Z.' • 

' 5 5 

- 


3 ' s * ? - 


8. i ♦ i : 

3 2 

* * 


r 


1 7 1 7 

♦ * 


s: • ♦ i- = 

2 1 7 

<* 


" 10. 1 1 . 
5 5 

J 

) 


o " 2b' 
ERIC . . 



* 


,3 X . I".- 

. u * 4 2 " 
1 

'-, • 
• ; . 


>TS\TB' . f - 


1 ^ 

. \ • - * 


— : " 


■* " » ~ 

13. 3| *. 2^'=- 

• 


20. 3 T - 

: ~ : *~ 


^ * 35 35 „ * 


21. 2 = • 

/ • * 


M. I x-2 - ' • 0 

2 8" * "* 

• • • - * * K ' - ' • . ' 


. 22. 3\ if = ' " " - • 


li. 1* ♦ 2* * 1 . 

■ - i 5 5 . ^ 


' 12.-', ^ ' ' . ' 

• * 3 - T * 1 • ' ■ 

♦ 

* 1 4 

i 


! . 1 ^3 

17. i + - = • 

.4,4. , 

' ERIC • ■ . ; N 


27 • t" • 

* * * 



